Application integration testing

ABSTRACT

Application testing is disclosed. A definition of a test to be performed on a subject application is received in a generic form not specific to the subject application. The test is performed by exchanging data with the subject application, as required to perform the test, using a test connector application associated with the subject application to do at least one of (1) convert an input data to be supplied to the subject application during the test from a generic data format not specific to the subject application into an application-specific data format associated with the subject application, if the application-specific data format is different than the generic data format and (2) normalize an output data received from the subject application in the application-specific data format into the generic data format not specific to the subject application, if the application-specific data format is different than the generic data format.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 12/217,088, entitled APPLICATION INTEGRATION TESTING filed Jun.30, 2008 which is incorporated herein by reference for all purposes,which is a continuation of U.S. patent application Ser. No. 10/943,778,now U.S. Pat. No. 7,421,621, entitled APPLICATION INTEGRATION TESTINGfiled Sep. 17, 2004 which is incorporated herein by reference for allpurposes, which claims priority to U.S. Provisional Patent ApplicationNo. 60/504,403 entitled APPLICATION INTEGRATION TESTING SYSTEM filedSep. 19, 2003 which is incorporated herein by reference for allpurposes.

FIELD OF THE INVENTION

The present invention relates generally to computers. More specifically,application integration testing is disclosed.

BACKGROUND OF THE INVENTION

Integrating disparate software systems often creates difficulties andcosts due to complexities in infrastructure. At an enterprise level,integrating software systems requires quality assurance (QA) testing,which is becoming increasingly complex. The increasing complexity ofsoftware systems requires that QA test systems and software also havegreater capabilities than predecessor applications.

For businesses, testing data systems requires thorough QA testing ofintegrated systems to ensure that data is properly routed betweenendpoints on the overall system. As legacy systems may often requireintegration with newer software systems, effective QA software must beable to test multiple processes and applications of varying types.Conventional QA integration testing software is often superficial as thecomplexity with testing integrated data systems often covers only theinput and output of data for particular graphical user interfaces (GUI).For example, conventional QA integration testing software packages thatare designed to be cross-platform compatible are often only capable oftesting for the proper input and output of data from system GUIs.Conventional QA integration testing software is typically designed forGUI testing or testing only one particular component application of anoverall larger system. Thorough testing of integrated software systemsrequires in-depth testing and evaluation across the entire platform andits individual and disparate components. This process can be costly ifconventional QA integration testing software is used, and may be costprohibitive if extensive custom coding is required. Moreover, for costreasons typical testing software systems at present provide only singlecomponent coverage via software tools, missing major portions ofintegrated systems, or is done manually with very marginal coverage.

A large part of the costs associated with conventional QA integrationtesting software are due to the labor-intensive nature of developmentand integration. Although conventional “off-the-shelf” packages areproduced by many software testing companies, specific integrationtesting with a particular system requires custom coding. Custom coding,in turn, requires platform-specific knowledge as well as testingapplication-specific knowledge that, when combined, requires significantlabor and time costs.

Thus, what is needed is a solution for thoroughly testing integratedsoftware packages that may be composed of disparate systems. There is aneed for a solution for testing integrated software systems thatautomates manual labor intensive parts of testing. Further, there is aneed for a solution for testing integrated software systems withoutincurring significant custom coding or labor costs.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 illustrates an application integration testing system, inaccordance with an embodiment of the present invention.

FIG. 2A illustrates a QA engine 200 of an application integrationtesting system, in accordance with an embodiment.

FIG. 2B illustrates a QA engine of an application integration testingsystem, in accordance with an embodiment.

FIG. 3 is a block diagram illustrating application testing asimplemented in one embodiment.

FIG. 4 is a block diagram illustrating application testing asimplemented in one embodiment.

FIG. 5 is a block diagram illustrating application testing asimplemented in one embodiment.

FIG. 6 is a flow chart illustrating a process used in one embodiment todesign and run a test in which runtime outputs are compared to baselinedata.

FIG. 7 is a flow chart illustrating a process used in one embodiment todesign and run a test in which runtime outputs are validated by applyingone or more validation rules.

FIG. 8 is a flow chart illustrating a process used in one embodiment torun a test.

FIG. 9 is a flow chart illustrating a process used in one embodiment toperform test operations.

FIG. 10 is a flow chart illustrating a process used in one embodiment toreceive data associated with a test.

FIG. 11 is a flow chart illustrating a process used in one embodiment toanalyze received test runtime output data that is to be compared tocorresponding baseline data.

FIG. 12 is a flow chart illustrating a process used in one embodiment toanalyze received test runtime output data that is to be validated usingone or more validation rules.

FIG. 13 is a flow chart illustrating a process used in one embodiment toidentify and run as a group a set of tests that satisfy one or morecriteria.

FIG. 14 is a block diagram illustrating an example of multiple testsinvolving data dependencies being run in sequence as a test chain, asimplemented in one embodiment.

FIG. 15 is a flow chart illustrating a process used in one embodiment todefine and run in sequence a chain of tests involving one or moredynamic data dependencies.

FIG. 16 is a flow chart illustrating a process used in one embodiment toperform in sequence the tests included in a test chain.

FIG. 17 illustrates a tree structure of a test project configurationfile used in one embodiment to configure a test platform and/orapplication to implement one or more of the techniques described herein.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess, an apparatus, a system, a composition of matter, a computerreadable medium such as a computer readable storage medium or a computernetwork wherein program instructions are sent over optical or electroniccommunication links. In this specification, these implementations, orany other form that the invention may take, may be referred to astechniques. In general, the order of the steps of disclosed processesmay be altered within the scope of the invention.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

FIG. 1 illustrates an application integration testing system, inaccordance with an embodiment. System 100 includes QA engine 102,connector application 104, data capture module 106, external environmentsetup/tear down module 108, and component call module 110.Cross-communication between QA engine 102, connector application 104,and data capture module 106 enables system 100 to test integratedsoftware applications such as an enterprise, platform, or scaledsoftware system. QA engine 102 is explained in greater detail below inconnection with FIGS. 2A and 2B. System 100 enables a test user,designer, or administrator to design, define, and run a test or set oftests.

External environment setup/tear down module 108 provides the initialsetup of the testing environment for system 100 with regard to externalapplications not directly associated with the test, such as by disablingalarms or other features associated with information technology resourcemonitoring and/or management applications such as Tivoli or OpenView.External environment setup/tear down module 108 also provides for teardown and “clean up” of the external test environment after testing hasbeen completed, such as by reactivating previously-disabled alarmsand/or other features. Component call module 110 provides communicationcapabilities for system 100 to call external (e.g., user-specific)components that are executed subject to rules or policies specified by,for example, a test designer or administrator.

Data is exchanged between system 100 and a subject application via testconnector application 104, in this embodiment. In general, various dataformats may be used including XML, HTML, or other common data typeformats. In one embodiment, one or more custom data type formats may bedefined. In one embodiment, a plug-in is used to define a custom datatype. Test connector application 104 communicates with test connectorsor adapters that are designed to provide a communication interface withvarious software types, packages, and platforms (e.g., SAP, PeopleSoft,Oracle, Vitria, Tivoli, file systems, FTP, etc.).

In one embodiment, a test connector application 104 (and/or one or moreinstances of the same connector) is provided for each subjectapplication (and/or instance thereof) with which the system 100 will berequired to communicate to perform a test and perform the job ofcommunicating with the application. The connector application 104understands the low level API/protocol to communicate with the subjectapplication with which it is associated and interface with QA engine 102and/or other components of system 100 in a standardized,non-application-specific way. In one embodiment, test connectorapplication 104 communicates with QA engine 102 asynchronously using XMLmessages. In one embodiment, each test connector application 104comprises a set of custom rules to interface with a given enterpriseapplication. In one embodiment, the custom rules are configured for thefollowing actions: connection properties; the publish properties; thesubscriber properties to use; conditions and duration to determinedynamically the behavior for when to connect/disconnect (e.g., how longa test connector communicates with an external application to publish orpoll/subscribe for data); application specific setup rules to set up thestate of an enterprise application; and application specific tear downrules to clean up the state of an enterprise application. In oneembodiment, test connector application 104 provides a data capturemodule. This data capture module has the functionality to extract datafrom an external application, which may be use for replaying as inputsor for data comparisons. In one such embodiment data capture module 106is an integral part of test connector application 104.

In one embodiment, there are three different types of test connectorapplication 104: source connectors, target connectors, and source-targetconnectors. Source connectors have the functionality of sending data toan application. In general, these types of connectors are used to be thesource of inputs into a test. Target connectors have the functionalityof retrieving data from an application. The actual process of how thedata is retrieved is specific to the application. The purpose of thistype of connector is to gather the outputs from an application and sendthem to QA engine 102 for analysis. Source-target connectors provide thecombined logic of a source and target connector. This type of connectoris associated with applications that are synchronous, like Web Servicesover HTTP.

In one embodiment, test connector application 104 is configured tocapture baseline data. In one embodiment, test connector application 104is configured to expose application-specific rules associated with anapplication involved in a test. For instance, in one embodiment adatabase test connector is configured to expose application-specificrules such as how long results should be polled for, how many recordsare expected, poll rate, etc. In one embodiment, an FTP test connectoris configured to expose FTP-specific information such asfiles/directories to delete in a remote FTP server; how long to poll;how long to wait before polling; poll rate; how many files to bedownloaded (optional); rules to consider when a file to be downloaded isdone (how long should the size/date of the file be fixed before it isconsidered complete); when a file to be downloaded is always changing,how long to wait before downloading the file once it is initially found;etc. Using test connectors developed for specific software packages,connector application 104 provides sampled and normalized data to system100. Data can be normalized to XML depending on the test application orleft in its original format, to perform integrated QA testing. Data maybe gathered from a variety of sources including application programminginterfaces (APIs), web services, databases (e.g., Oracle), or otherplatform enterprise applications (e.g., SAP, PeopleSoft, etc.).Depending on the application with which a connector is associated, datamay be received and/or provided by the test connector application in theform of XML or other data, a file, and/or a message or even multi-part(complex) format, a data type with many parts of other data types. Inone embodiment, QA engine 102 communicates with test connectorapplication 104 via XML messaging. Data capture module 106 is configuredto capture baseline data. In one embodiment, data capture module 106captures data based on a user-defined test case and obtains the datadesignated by the user from locations specified by the user.

In one embodiment, test connector application 104 conforms to a standardpromulgated by a competent authority or body to specify a format to beused to create standard and/or custom (e.g., plug-in) test connectorapplications. In one embodiment, the system 100 comprises a workflow or“business process management” engine. In one embodiment, the system 100is implemented in a J2EE application server.

In one embodiment, system 100 comprises a graphical user interface(GUI), not shown in FIG. 1, to enable a user to create, update, or edittest projects, configure data capture, etc. In one embodiment, system100 comprises an administration GUI, not shown in FIG. 1, to facilitatemanagement of the system 100, e.g., by adding/removing components (e.g.,test connector applications, data types, users, etc.). In oneembodiment, system 100 comprises a scheduling GUI, not shown in FIG. 1,to schedule test run times, define data or other dependencies, identifya server in which a test should be run, etc. In one embodiment,dependencies between test projects (and/or other test nodes, describedmore fully below) may be defined.

FIG. 2A illustrates a QA engine 200 of an application integrationtesting system, in accordance with an embodiment. In one embodiment, QAengine 200 is used to implement QA engine 102 of FIG. 1. QA engine 200includes correlator 202, test analyzer 204, XPATH library module 206,internal test environment module 208, and filter module 210. Filtermodule 210 also includes inclusion filter 212 and exclusion filter 214.For data communication between QA engine 200 and other components, suchas test connectors used by QA engine 200 to exchange data with one ormore applications being tested, channel queue 216 is shown withsampler/mediator 218. In one embodiment, there is a separate testconnector application 104 for each different application with whichinteraction is required to perform a test and/or possibly multipleinstance of the same connector. In one such embodiment each testconnector and/or instance thereof has associated with it a channel suchas channel 216.

In this example, QA engine 200 provides logic capabilities for variousfunctions performed by application integration testing system 100. Inone embodiment, correlator 202 correlates runtime data from anintegrated application undergoing testing with corresponding baselinedata. In one embodiment, this correlation is performed by normalizingthe runtime data to a common format (e.g., XML) and applying apredefined correlation rule or process to identify the correspondingbaseline data. Correlation may be based on data fields configured aspart of the control parameters for system 100 (e.g., “date,” “time,”“number,” “type,” etc.). In one embodiment, a test engineer or otheruser or process provides an identification of or path to the baselinedata corresponding to an element of runtime data, e.g., using an XPATHexpression. In one embodiment, correlator 202 may be configured toassume data is provided as output at runtime in the same sequence as itwas provided when the baseline data was captured, and correlation isbased on this assumption. In some embodiments, QA engine 202 may performone or more tests or operations that do not involve or requirecomparison to baseline data. In one such embodiment, QA engine 200 maybe configured to validate runtime data against one or more applicablevalidation rules, as described more fully below. In an embodiment, test,or operation in which a comparison to baseline data is not required orperformed, correlator 202 plays no role.

In one embodiment, test analyzer 204 provides a comparison feature ofbaseline data to run-time output data to ensure proper operation of anintegrated application undergoing testing. In one embodiment, testanalyzer 204 may be configured to compare only selected data fields orelements so as to ignore differences that would not indicate an error,such as different time stamps. In an embodiment, test, or operation inwhich analysis other than comparison to baseline data is involved, testanalyzer 204 may be configured to validate runtime data by applying oneor more applicable validation rules. In one embodiment, validation rulesmay be applied to runtime data using a validator module not shown inFIG. 2A. XPATH library module 206 provides reference data in the form oflibraries in order to aid the performance of correlator 202, testanalyzer 204, and filter module 210. In one embodiment, XQUERY oranother alternative and/or successor to XPATH is used. XPATH librarymodule 206 may be a library storing XML path language to assist in theparsing of XML-based data. Other libraries may be included within XPATHlibrary module 206 that aid in parsing formats other than XML.

Internal test environment module 208 provides several capabilities thatenable testing to occur by performing tasks required to set up withrespect to the application to be tested the initial conditions requiredfor testing and/or to clean up with respect to the tested applicationafter testing. For example, removal of persistent XML data associatedwith an application to be tested prior to testing may be performed byinternal testing environment module 208 in order to ensure thatinaccurate test results do not occur.

In order to provide accurate and thorough testing of various softwarecases (e.g., one input to one output, one input to many output, manyinputs to one output, many inputs to many outputs), filter module 210 isprovided in addition to correlator module 202. An inclusion filter 212ensures that the proper data inputs are provided to QA engine 200.Likewise, exclusion filter 214 ensures that unnecessary outputs are notused by QA engine 200. In one embodiment, correlator module 202 providesthe matching capabilities to determine which runtime output goes withwhich baseline output in order to be compared. In one embodiment, thisensures that runtime outputs are correlated with the correctcorresponding baseline data for comparison, even if the runtime outputsarrive in different order from test run to test run for the same input,such as may occur, for example, in asynchronous enterprise applications,like MQ, JMS, VITRIA, TIBCO and other message based systems. Externalcommunication between QA engine 200 and an integrated applicationundergoing testing may be provided by channel queue 216, which retrievesdata from the integrated application via sampler/mediator 218.

In some embodiments, QA Engine 200 works with applications that generatedata in the form of messages (e.g., JMS, Web Services, IBM MessageQueue), as well as XML, EDI, ASCII, binary data, and other data.Additionally, QA engine 200 works with applications that generate filesas output or use files as input data. In one embodiment, test analyzer204, correlator 202, and filter module 210 (including inclusion filter212 and exclusion filter 214) include logic for handling different dataformats, in order to ensure proper handling of tested applications thatare not able to generate XML output. In other words, testing flexibilityis included in these components in order to handle data that is not XMLand allow QA engine 200 to operate. In one embodiment, test analyzer 204comprises several comparison engines. In one such embodiment, testanalyzer 204 comprises the following comparison engines: XML Compare,ASCII Compare, Binary Compare, EDI Compare, JMS Compare, and MessageCompare. In one embodiment, additional custom data type comparisonmodules (e.g., plug-ins) may be included in QA engine 200 to supportadditional custom rules for comparing and/or validating a given datatype. Like specific data type comparison/validation engines, suchadditional custom data type comparison modules include counterparts forcorrelation and filtering, which in one embodiment also are added forcustom data types via a plug-in or other mechanism. In some embodiments,it is not necessary to convert ASCII, binary or other forms of data inorder for QA engine 200 to operate. QA Engine 200 can use ASCII, binary,or other forms of data for base comparisons, but data in ASCII, binary,or other forms may be limited in terms of the fidelity of thecomparison/validation. ASCII, binary, or other forms of data may be inany format, and QA engine can determine differences between runtime datain a variety of formats and corresponding baseline data. In someembodiments, custom data type plug-in objects may enable specialcomparison, correlations, filtering, transformation (e.g., decryption),and other functions using ASCII data, binary data, and/or data in custom(non-standard) data type formats. However, if data is output in astandard data type format (e.g., in one embodiment, XML, ASCII, EDI, orJMS), then no custom plug-in is necessary for the operation of QA engine200. In one embodiment, a custom data type plug-in is required only toenable processing of many inputs to many outputs or one input to manyoutputs test case, where the data arrives asynchronously and/or is knownto vary in order from test run to test run. In one embodiment, a customdata type plug-in is required if the data is not of a standard type andthe fidelity of the comparison/validation rules is required to go downto the field level. In such an embodiment, the data must be converted,e.g., using a custom data type plug-in, to and from XML and/or anotherform in which such field level analysis may be performed with therequired fidelity.

In one embodiment, data may be of a complex data type. For example, insome cases complex data types may include metadata and/or multipleparts, one or more of which may comprise data of different types. In oneembodiment, a complex data type may be a standard complex data type thatthe QA engine 200 is configured to recognize and process, e.g., byparsing the complex data to extract the component parts required forprocessing. In one embodiment, custom complex data types may be defined,including without limitation by defining custom comparison, correlation,filtering, and/or transformation plug-ins for such custom complex datatypes.

QA engine 200 provides capabilities for designing tests in an enterpriseintegrated environment. Unlike conventional QA techniques, QA engine 200is not restricted in the number of enterprise applications that can beused to design, implement, perform, or analyze a test. N number ofapplications may be operated upon by QA engine 200 and system 100.

In one embodiment, QA Engine 200 is configured to perform functionaltesting of various components, regardless of system design, in order todetermine whether the business functionality provided is executingproperly. QA engine 200 also determines in one embodiment when a systemchange that results in an error (e.g., operation, environment,configuration, etc.) occurs in an application. If so, QA engine 200 canprovide a test result summary that precisely determines where the errorbegan and its location in the tested applications involved in anintegrated test.

In one embodiment, QA Engine is designed and may be configured to handleone or more of various types of QA tests, including, but not limited tothe following:

a. Regressional QA. By recording a baseline data set for input andoutputs, system 100 can performed tests using the baseline input andcompare test results (i.e., outputs generated at test run time) againstthe baseline outputs. Baseline data can be referenced from the testdefinition (e.g., an XML configuration file or other specified file) orin a manner independent of location (e.g., file system, database, LDAPserver, etc.).

b. Functional QA. Allow the test designer to apply n number ofvalidation rules to validate the runtime output data from an applicationbeing tested. In one embodiment, the validation rules that can beapplied against one or more fields; can be any of the following:permitted/expected range of values; list of permitted/expected values;regular expression; mathematical operator-defined (e.g., greater than X,less than Y, not equal to Z); stream validation using text searchengines like Lucene; date validation rules for any fields (including inone embodiment date range, date operation, date in a list of knowndates, and user-defined date rules with relative date parameters, e.g.,a rule requiring that the value of a particular date field be equal tothe current day's date whenever the test is run); or custom JAVA class(or similar) to validate a specific field or set of fields.

c. Performance Testing. If a specific time frame has been provided inwhich to execute a test, system 100 and QA engine 200 ensure that afunctional QA test is completed within a specified time frame. In oneembodiment, if a specified time frame is exceeded, a failure is noted,and the test is either continued to completion or is stopped based on auser's preference, rule, or policy, as defined in the test definition.In other embodiments, other criteria may be used to indicate successfulor failed test results.

d. Stress Test. QA engine 200 can run a functional QA test n number oftimes and, for every run, apply rules or policies specified forfunctional QA testing.

e. Robustness Testing. These types of tests include those tests wherethe test objective is to determine how a particular application or anintegrated set of applications respond to different types of inputs.Given a baseline input or set of inputs, a test designer, using QAengine 200 and system 100, can vary any of the inputs into system 100,prior to running tests. The following types of input changes areexamples of what a test designer, user, or administrator may define in atest (or test project comprising multiple tests) in one embodiment:

-   -   i. Change the value of an input parameter between a min and max        value, incrementing with value n, for a given variable type.    -   ii. Change the value of an input parameter with a set of values,        where the test designer selects particular values (e.g., gold,        platinum, silver, etc.).    -   iii. Change the value of an input parameter by QA engine 200        generating a random value, between a min and max value for a        given variable type.    -   iv. Change the value of an input by varying with values from        data located in a designated file/database table.    -   v. Change the value of an input by specifying a custom java        class, which will provide a list of values.

f. Test Chaining In a group of tests in which an input required to beprovided for a one test is dependent dynamically on the value of a testruntime output generated by another test, the system may be configuredto run the tests sequentially and update the dynamically dependent inputprior to running the test with which the input is associated.

g. Test Include. A specified location (e.g., file directory, database)may be searched for test-associated files (e.g., test definition files,test projects comprising multiple test nodes, etc.) that satisfyspecified criteria (e.g., all test projects in a particular directory;all test projects within a directory that satisfy a specified regularexpression, all test projects that appear on a provided list of testprojects, etc.).

The above paragraphs a-g describe examples of various types of teststhat can be performed in one embodiment, but in other embodiments othertests not listed above may be performed. In one embodiment, the userdefines the tests to be performed by system 100 via one or more XML orother configuration file, referred to in one embodiment as a “testproject” configuration file. In this manner, the system can performtests throughout an integrated set of applications without writing anycustom code (e.g., plug-in) in java, c, c++, perl, etc.

FIG. 2B illustrates a QA engine of an application integration testingsystem, in accordance with an embodiment. In this example, QA engine 200and its components are similar to those discussed in connection withFIG. 2A. In this embodiment, converter engine 220 is also included withQA engine 200. Converter engine 220 allows, for example, a test designerto convert proprietary data in various formats (e.g., ASCII, Binary,etc.) to XML. This normalized data is then provided to QA engine 200 toenable consistent testing of all tested application components. In oneembodiment, converter engine 220 converts custom data type formats,defined as described above, to XML, e.g., to facilitate field-levelcomparison and/or validation.

FIG. 3 is a block diagram illustrating application testing asimplemented in one embodiment. In the example shown, a baseline input302 is provided to a tested application (or set of applications) 304.The tested application(s) 304 generate test run time outputs 306 and308. In one embodiment, the runtime outputs 306 and 308 are supplied toa QA engine for analysis as described herein. In one embodiment, theruntime outputs 306 and 308 are correlated with and compared tocorresponding baseline values. If the runtime values deviate from thebaseline values, responsive action (e.g., notify a user that the testresult is “fail”) is taken.

FIG. 4 is a block diagram illustrating application testing asimplemented in one embodiment. A QA engine uses a test connector 404 toexchange data with a tested application (or set of applications) 406. Inone embodiment, test connector 404 is an instance of a test connectorapplication such as described above in connection with test connectorapplication 104 of FIG. 1. QA engine sends a command (or baseline inputdata) 408 to connector 404, in response to which test connector 404provides a test input 410 to application 406. In one embodiment, testconnector 404 converts baseline input data associated with 408 into aformat suitable for application 406 and provides the data to application406 via a communication link. Application 406 performs operations basedon input 410 and generates test run time outputs 412 and 414. Runtimeoutputs 412 and 414 are provided to (and/or obtained by, depending onthe application and/or configuration) test connector 404, converted (ifnecessary) and normalized (if necessary) by test connector 404 into aformat suitable for use by QA engine 402 (e.g., XML, ASCII, etc.), andprovided to QA engine 402 as normalized test runtime outputs 416 and418.

FIG. 5 is a block diagram illustrating application testing asimplemented in one embodiment. In this example, two applications areinvolved in the test. A QA engine 502 provides a command (or baseline orother input data) 504 to a first test connector 506, which converts thedata into input data 508 provided to a first application 510 in a formsuitable for use by first application 510. First application performsoperations based on input data 508 and generates intermediate outputdata 512, which is provided as input to a second application 514. Secondapplication 514 in turn operates on input 512 and generates test runtimeoutputs 516 and 518. Test runtime outputs 516 and 518 are provided to(or obtained by, e.g., by polling) a second connector 520, which in turnconverts and normalizes the test runtime outputs 516 and 518, asrequired, and provides them to QA engine 502 as normalized test runtimeoutputs 522 and 524. QA engine 502 in one embodiment compares normalizedtest runtime outputs 522 and 524 to corresponding values in a set ofbaseline data, as described herein.

In the examples shown in FIGS. 3-5, one input generates many(specifically in these examples two) outputs, but the approachesdescribed herein may be used in other situations as well, e.g., manyinputs to many outputs, one input to one output, many inputs to oneoutput, etc. In addition, the examples shown in FIGS. 3-5 are forpurposes of illustration only, and are presented solely to providespecific, concrete examples of the application testing approaches andtechniques described herein, which are applicable generally and can beused and/or configured as applicable to test any application orcombination thereof in any configuration, within the complete control ofa test engineer or other user. In one embodiment, all that is requiredfor the techniques described herein to be used to test a particularapplication or set of applications is a configuration file that definesthe tests to be performed, the location of the components with which thetesting system will exchange data, and one or more connectorapplications or modules capable of exchanging data between the testingsystem and the application(s) with which the testing system will berequired to exchange data to perform the test.

FIG. 6 is a flow chart illustrating a process used in one embodiment todesign and run a test in which runtime outputs are compared to baselinedata. In 602, the test is designed. In one embodiment, a test engineeror other user identifies the components, processes, and/or functions tobe tested and determines the data inputs and outputs to be controlled,monitored, and/or analyzed to determine if the subject components,processes, and/or functions are performing properly. In one embodiment,602 includes preparing an XML configuration file that defines thecomponents to be tested and the tests to be performed. In 604, requiredtest system components are installed and/or configured, as required. Inone embodiment, 604 may include installing and/or configuring anapplication-specific test connector application for each applicationwith which the test system will be required to exchange data. In oneembodiment, a user is provided with a library of application-specifictest connectors each associated with one or more specifically supportedapplications and 604 includes a user selecting from the library andinstalling one or more application-specific test connectors associatedwith the application(s) to be tested. In one embodiment, the requiredapplication-specific test connector(s) may be selected and installedautomatically, e.g., by logic configured to select and install therequired test connectors based on information provided by a userregarding an application to be tested and/or information discoveredautomatically. In one embodiment, 604 may include identifying, defining,and/or configuring communication channels between a QA engine and one ormore connectors and/or between one or more test connectors and thesubject application with which they are associated. In 606, baselinedata is established and validated. In one embodiment, a test casedefined by a test engineer and/or other user is run and the inputsprovided and outputs generated during the running of the test case arecaptured and stored as baseline data. The baseline data is thenvalidated, to ensure the data represents the normal and expectedoperation of the application(s) to be tested. At 608, the test is run atrun time. For example, the test may be run periodically to ensure thecontinued error-free operation of the subject application(s). Or, thetest may be run after a change to the subject application(s) and/or thehost systems on which they are installed, such as installation of asoftware patch or upgrade. The results of the test are evaluated at 610.If the subject application failed the test, at 612 responsive action istaken. In one embodiment, the responsive action comprises generating andsending a notification. In some embodiments, 612 may comprise furtherprocessing to pinpoint if possible the cause of failure. If the subjectapplication performed satisfactorily, at 614 a “pass” result is reportedand the process ends. In one embodiment, both 612 and 614 includewriting test results to a location (e.g., file, directory, other userspecified location). In one embodiment, test results are created in XMLor any data format suitable for making such results available to a user,such as HTML, PDF, etc. In one embodiment, errors and/or failuresidentified during testing are associated in the test results with a testconnector application with which the result is associated, therebyfacilitating localization and troubleshooting of the cause of the erroror failure. In one embodiment, analyzing test runtime data in XMLfacilitates more precise localization of the cause of an error orfailure.

FIG. 7 is a flow chart illustrating a process used in one embodiment todesign and run a test in which runtime outputs are validated by applyingone or more validation rules. In 702, the test is designed. In 704,required test system components are installed and/or configured, asrequired. At 706, validation rules to be used to validate test runtimedata are received and/or determined. In one embodiment, the validationrule(s) may be defined by a test engineer and/or other user, e.g., in anXML or other configuration file. In one embodiment, a test case is runand the testing system is configured to capture one or more runtimeoutputs generated during a run of the test case and derive from the testcase runtime outputs one or more validation rules to be applied tocorresponding output data generated during subsequent test runs. At 708,the test is run at run time. In one embodiment, one or more elements ofoutput data generated at runtime are validated using one or more of therules received and/or determined at 706. The results of the test areevaluated at 710. If the subject application failed the test, at 712responsive action is taken if configured. In one embodiment, theresponsive action comprises generating and sending a notification ifconfigured. In some embodiments, 712 may comprise further processing topinpoint if possible the cause of failure. If the subject applicationperformed satisfactorily, at 714 a “pass” result is reported and theprocess ends. In one embodiment, both 712 and 714 include writing testresults to a location (e.g., file, directory, other user specifiedlocation).

FIG. 8 is a flow chart illustrating a process used in one embodiment torun a test. In one embodiment, the process of FIG. 8 is used toimplement 608 of FIG. 6. In one embodiment, the process of FIG. 8 isused to implement 708 of FIG. 7. At 802, any actions required to set upthe testing environment are performed. In one embodiment, such actionsmay include creating, storing, changing, and/or deleting one or moreitems (e.g., data, database record, file, etc.); starting or stopping anapplication program and/or processing required to be in a particularstate at test time; etc. In 804, the test operations that comprise thetest are performed. Depending on the configuration and the design andimplementation of the particular test being run, the test operations mayinclude in one embodiment providing (via a test connector) one or moreinputs to one or more subject applications (i.e., applications that arebeing tested); receiving output (via a test connector) from one or moresubject applications; and/or analyzing received runtime data (e.g., bycomparing it to corresponding baseline data, validating it using one ormore validation rules, etc.). At 806, post-test clean up is performed,after which the process ends. In one embodiment, such post-test clean upmay include deleting data records, files, and/or other items generatedduring the testing; restoring data records, files, and/or other items tothe state (e.g., value) in which they were before the test was run; etc.

FIG. 9 is a flow chart illustrating a process used in one embodiment toperform test operations. In one embodiment, the process of FIG. 9 isused to implement 804 of FIG. 8. At 902, information required to performthe next test operation event is retrieved and processed. If the eventrequires that one or more inputs be provided to one or more subjectapplications (904), the required input data is retrieved and sent to thesubject application(s) in 906 and the processing advances to the nexttest operation event at 902. Otherwise, test runtime output data isreceived at 908. In 910, the test runtime output data is stored locallyfor later analysis and/or use (e.g., to update an input to be providedto a subject application in a later event and/or test) and/or isanalyzed in real time, depending on the embodiment and/or configuration,such as by comparing the runtime output to corresponding baseline dataand/or validating the runtime output using one or more applicablevalidation rules. If the test operation event just completed was thelast operation of the test (912), at 914 any required analysis ofreceived test runtime output data is completed and the test results arereported. Otherwise, the processing advances to the next test operationevent at 902, after which the process repeats for each test operationevent until all operations comprising the test have been completed. Inother embodiments, one or more test operation events comprising a testmay involve processing other than and/or in addition to providing inputsand/or receiving and analyzing test runtime output data, e.g., startingor stopping a system, application, or process required to be started orstopped at prescribed points in the test, etc.

FIG. 10 is a flow chart illustrating a process used in one embodiment toreceive data associated with a test. In one embodiment, the process ofFIG. 10 is used to implement 908 of FIG. 9. At 1002, it is determinedwhether the received data is a recognized data type. If so, the receiveddata is normalized at 1004. In one embodiment, the preferred data formatis XML and all data that is not in XML is converted to XML if possible.In one embodiment, complex data types, e.g., JMS messages or others thatmay comprise data of two or more different types, are normalized byconverting each portion, to the extent possible, to a common XML format.In one embodiments, a test platform is configured, e.g., by installing atesting application so configured, to recognize one or more data typesin formats other than XML, e.g., EDI, and convert such data into XMLdata that conforms to a schema associated with the test platform and/orapplication. If the received data is determined at 1002 to be somethingother than a recognized standard data type, at 1006 it is determinedwhether the data is of a defined custom data type. In one embodiment, auser may define one or more custom data types by defining a custom datatype plug-in (implemented in Java, or similar language) that enables thetest platform and/or application to recognize the custom data format andconvert data received in that format to XML data that conforms to aschema and/or custom data type definition associated with the testplatform and/or application. If the received data is determined at 1006to be of a defined custom data type, at 1008 the JAVA conversion class(or other plug in, depending on the embodiment) associated with thecustom data type is used at 1008 to normalize the data to the preferredformat, e.g., XML. If the data is neither of a recognized standard typenor a defined custom type, in 1010 analysis of the data is limited tolow level comparison, e.g., ASCII or binary data comparison.

FIG. 11 is a flow chart illustrating a process used in one embodiment toanalyze received test runtime output data that is to be compared tocorresponding baseline data. In one embodiment, the process of FIG. 11is used to implement at least in part 910 and/or 914 of FIG. 9,depending on the embodiment. At 1102, the received (and, if applicable,normalized) test runtime output data is filtered. In one embodiment, thefiltering comprises identifying in and extracting from the received dataone or more data elements to be analyzed (i.e., an inclusion filter). Inone embodiment, the filtering comprises excluding from analysis one ormore data elements. In one embodiment, one or more exclusion filters maybe used to exclude from analysis data elements that may vary from run torun from a corresponding element of baseline data without such deviationfrom the baseline indicating an error or other problem, e.g., atimestamp, invoice number, and/or other values that may vary validlywith time or successive iterations of the process(es) being tested. In1104, the data identified for inclusion and/or not excluded in 1102 iscorrelated with the baseline data that corresponds to it. In oneembodiment, the correlation performed at 1104 includes using one or moreXPATH and/or XQUERY expressions defined by a user to find the baselinedata that corresponds to each element of received test runtime data. Inone embodiment, it is assumed (or the test platform and/or applicationmay be configured by a user to assume) that the test runtime data isreceived in the same sequence as the baseline data, and the correlationis based on this assumption. The received data is then compared at 1106with the corresponding baseline data. If the test runtime data isdetermined at 1108 to not be the same as the corresponding baseline data(or to not be sufficiently the same as, depending on the data and theembodiment), it is concluded in 1110 that the result for the test (orportion thereof) with which the data is associated is “fail”. If thetest runtime data is determined at 1108 to be the same as thecorresponding baseline data, it is concluded in 1112 that the result forthe test (or portion thereof) with which the data is associated is“pass”.

FIG. 12 is a flow chart illustrating a process used in one embodiment toanalyze received test runtime output data that is to be validated usingone or more validation rules. In one embodiment, the process of FIG. 12is used to implement at least in part 910 and/or 914 of FIG. 9,depending on the embodiment. At 1202, the received (and, if applicable,normalized) test runtime output data is filtered, e.g., as describedabove in connection with 1102 of FIG. 11. In 1204, the validationrule(s) to be applied to the data identified for inclusion and/or notexcluded in 1202 is/are determined. The applicable validation rule(s)is/are applied to the data at 1206. If the test runtime data isdetermined at 1208 to not be valid, it is concluded in 1110 that theresult for the test (or portion thereof) with which the data isassociated is “fail”. If the test runtime data is determined at 1208 tobe valid, it is concluded in 1212 that the result for the test (orportion thereof) with which the data is associated is “pass”.

In one embodiment, a test may involve two or more integrated enterpriseapplications. In such an embodiment, a separate application-specifictest connector application (and/or instance thereof) is used tocommunicate with each subject application involved in the test. In oneembodiment, if an error and/or test failure occurs (e.g., runtime outputdata does not match corresponding baseline date, or runtime output datadoes not satisfy an applicable validation rule), the error iscategorized base on the subject application that caused the error (e.g.,the subject application that generate test runtime output data thatfailed a comparison to baseline and/or application of a validationrule). In one embodiment, this categorization is made by associating theerror with the application-specific test connector application used tocommunicate with the subject application associated with the error. Inone embodiment, certain failures may be categorized in a manner notspecific to any one subject application, such as failure of the overalltest to be completed within a prescribed time in a performance test.

In one embodiment, an “error” associated with a subject applicationand/or test connector application may be associated with one or more“failures”, where a “failure” is a single “fail” event, such as anactual difference between a field in the baseline and runtime output,failure to satisfy a particular validation rule, etc. This approachenables a person analyzing the results to know what failures are groupedtogether within a given error and more quickly localize and analyze allthe problems with a given set of test runtime output data.

In some cases, an error may not have any failures associated with it,such as when a connection cannot be established to a particular subjectapplication. In one embodiment, the test can be configured to stoprunning after a single failure event, e.g., where a test designer onlycares to know that at least one comparison or validation failed and doesnot need to know whether others also would have occurred had the testcontinued to run.

FIG. 13 is a flow chart illustrating a process used in one embodiment toidentify and run as a group a set of tests that satisfy one or morecriteria. In one embodiment, the set of tests that satisfy the criteriamay include zero, one, or many tests, depending on the configuration,the criteria used, and the tests that have been defined. At 1302, anindication is received that tests meeting one or more specified criteriashould be run as a group. In one embodiment, the tests may include testsdefined previously by a user and stored, e.g., in the form of an XMLconfiguration or test definition file, in one or more file systemdirectories, a database, and/or other location, repository, orstructure. In one embodiment, the criteria may include an identificationof the location(s) to be searched, e.g., all test files (e.g., tests,test groups, test projects) found in a specified directory. In oneembodiment, additional and/or different criteria may be specified, suchas test files that match a regular expression, test files that appear ina list of files to be included, etc. At 1304, tests (or test groups,test projects, etc.) that satisfy the specified criteria are found. Inone embodiment, 1304 comprises searching a specified location (e.g.,directory) for tests that satisfy the specified criteria (e.g., all testfiles, those that match a specified regular expression, those thatappear on a provided list, etc.). At 1306, the tests determined tosatisfy the criteria are run. In one embodiment, a user may specify thatthe tests be run sequentially, concurrently, and/or in groups of aspecified size (e.g., four at a time). In this manner, groups of relatedtests may be run in a convenient and efficient way.

FIG. 14 is a block diagram illustrating an example of multiple testsinvolving data dependencies being run in sequence as a test chain, asimplemented in one embodiment. In the example shown, a QA engine 1402 isconfigured to perform a first test by providing a command (or datainput) 1404 to a first test connector 1406. First test connector 1406 isconfigured to provide to a first application 1410 an input 1408 in aformat appropriate for the first application 1410. The first application1410 processes the input and generates a test runtime output 1412, whichis provided to (or obtained by) the first test connector 1406. The firsttest connector 1406 normalizes the data into a format usable by QAengine 1402 and provides the normalized test runtime output 1414 to QAengine 1402. QA engine 1402 is also configured in the example shown toperform a second test by providing a second command (or data input) 1416to a second test connector 1418, which is configured to in responseprovide a second data input 1420 to a second application 1422. Thesecond application 1422 processes the input 1420 and generates tworuntime outputs 1424 and 1426, which are normalized by a secondconnector 1418 prior to being provided to QA engine 1402 as normalizedoutputs 1428 and 1430.

In the example shown in FIG. 14, it is assumed for purposes ofillustration that the value of the second input 1420 is dynamicallydependent on the value of the test runtime output 1412 of firstapplication 1412, as generated during the running of the first test.That is, in order for the second test to be run properly and the resultsunderstood and analyzed properly, it is necessary for the first test tobe run and the input used to run the second test updated based on aresult of the first test (e.g., an output value generated during therunning of the first test) prior to running the second test. In oneembodiment, the QA engine and/or an associated component or process isconfigured to run the first test and update prior to running the secondtest an input value associated with the second test based at least inpart on an output generated during the running of the first test.

FIG. 15 is a flow chart illustrating a process used in one embodiment todefine and run in sequence a chain of tests involving one or moredynamic data dependencies. At 1502, a definition of the tests to beincluded in the test chain and any applicable data dependencies isreceived. In one embodiment, the definition is received in the form ofan XML configuration or test definition file. In one embodiment, datadependencies are defined by defining an update rule comprising an XPATHexpression identifying a path to an output data value to be used toupdate a specified input value associated with the update rule prior torunning a test with which the specified input value is associated. At1504, an indication is received that the test chain (i.e., the testscomprising the chain defined in 1502) should be run. At 1506, the testsassociated with the test chain are run in sequence, using the outputfrom one or more tests in the chain, as applicable, to update one ormore inputs for tests to be run later in the sequence prior to runningsuch later tests.

FIG. 16 is a flow chart illustrating a process used in one embodiment toperform in sequence the tests included in a test chain. In oneembodiment, the process of FIG. 16 is used to implement 1506 of FIG. 15.At 1602, data associated with the next test in sequence is retrieved andprocessed. In one embodiment, in the first iteration of the processshown in FIG. 16 data associated with the first test in the chain isretrieved and processed at 1602. In 1604, it is determined whether thenext test to be performed is associated with any defined dynamic datadependency, i.e., whether any input associated with the test is requiredto be updated, prior to performing the test, based on an outputgenerated by a previously-run test. If the test is associated with oneor more data dependencies, any inputs required to be updated are updatedat 1606 prior to running the test at 1608. If no input is required to beupdated prior to running the test, 1606 is bypassed and the test is runat 1608. In 1610, it is determined whether the test just run is the lasttest in the chain. If so, the process ends at 1612. Otherwise, theprocess returns to 1602 and the next test in the chain is processed,after which the process of FIG. 16 repeats for each test in the chainuntil the last test has been performed.

FIG. 17 illustrates a tree structure of a test project configurationfile used in one embodiment to configure a test platform and/orapplication to implement one or more of the techniques described herein.In one embodiment, the configuration file illustrated in FIG. 17comprises an XML file. In one embodiment, the configuration file of FIG.17 comprises an input into QA engine 200 or integration testing system100. In the example shown, a user defines a test project comprising oneor more test nodes 1704, represented in FIG. 17 by test nodes 1 throughn. In one embodiment, each node may represent a test, a group of tests,a test chain, a set of tests that satisfy criteria associated with thenode, a stress or robustness (hardness) test such as described above, orany other test or related project. The structure illustrated in FIG. 17allows a test designer to group multiple tests within a group, forexample, using applied performance and/or running mode (e.g., sequentialor parallel), which indicates how tests are run within a group. Otheroptions for grouping tests may also be included. In one embodiment, eachof the test nodes may include one or more children nodes and/or one ormore tests. In the example shown, test node 2 includes a plurality ofchildren test nodes 1706, represented in FIG. 17 by test nodes 2.1 to2.i, and a plurality of tests 1708, represented in FIG. 17 by test leafs2.1 to 2.j. In one embodiment, each node may have one or more userconfigurable properties associated with it, including as applicable andwithout limitation, properties defining a maximum time for operationsassociated with the node to be completed and whether the operationsshould be continued to completion even if the maximum time is reached;indicating whether operations associated with the node, e.g., thosespecified in children test nodes and/or test leafs associated with thenode, should be performed sequentially, concurrently, or in groups of aspecified size (e.g., four at a time); specifying a location to whichresults data should be stored; etc. In one embodiment, one or morecustom types of test node may be defined, e.g., via a user-definedplug-in.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

1. A method of testing an application, comprising: receiving adefinition of a test to be performed on a subject application; andperforming the test; wherein performing the test comprises exchangingdata with the subject application, as required to perform the test,using a test connector application associated with the subjectapplication to do at least one of (1) convert an input data to besupplied to the subject application during the test from a generic dataformat not specific to the subject application into anapplication-specific data format associated with the subjectapplication, if the application-specific data format is different thanthe generic data format and (2) normalize an output data received fromthe subject application in the application-specific data format into thegeneric data format not specific to the subject application, if theapplication-specific data format is different than the generic dataformat.
 2. A method as recited in claim 1, wherein the subjectapplication comprises two or more integrated subject applications andperforming the test comprises using a separate test connectorapplication specific to each of at least a subset of said two or moreintegrated subject applications to do with respect to the integratedsubject application with which it is associated at least one of (1)convert an input data to be supplied to the subject application duringthe test from a generic data format not specific to the subjectapplication into an application-specific data format associated with thesubject application, if the application-specific data format isdifferent than the generic data format and (2) normalize a test runtimeoutput data received from the subject application in theapplication-specific data format into the generic data format notspecific to the subject application, if the application-specific dataformat is different than the generic data format.
 3. A method as recitedin claim 1, wherein performing the test comprises correlating a testruntime output data received during the test with a correspondingbaseline data associated with the test.
 4. A method as recited in claim1, wherein performing the test comprises comparing a test runtime outputdata received during the test to a corresponding baseline dataassociated with the test.
 5. A method as recited in claim 1, whereinperforming the test comprises comparing a test runtime output datareceived during the test to a corresponding baseline data associatedwith the test and taking responsive action if it is determined the testruntime output data is different than the corresponding baseline data.6. A method as recited in claim 1, further comprising generating a setof baseline data associated with the test by running a baselineiteration of the test using a predefined test case and capturing one ormore elements of baseline data generated during the baseline iterationof the test.
 7. A method as recited in claim 1, further comprisingderiving a validation rule associated with the test by performing abaseline iteration of the test using a predefined test case, capturingone or more elements of baseline data generated during the baselineiteration of the test, and deriving the validation rule based at leastin part on at least a subset of the one or more elements of capturedbaseline data.
 8. A method as recited in claim 1, wherein the testconnector application is one of a plurality of test connectorapplications, each specific to a corresponding subject application forwhich testing is supported.
 9. A method as recited in claim 1, whereinthe test connector application is selected from a plurality of connectorapplications, each specific to a corresponding subject application forwhich testing is supported.
 10. A method as recited in claim 1, whereinthe test comprises a first test, the subject application comprises afirst subject application, the definition comprises a first definition,and the method further comprises: receiving in the generic form a seconddefinition of a second test to be performed on a second subjectapplication; and receiving an update rule defining a dynamic dependencybetween a dependent input to be supplied to the second application aspart of the second test and a corresponding output generated by thefirst subject application when the first test is run and requiring thatthe value of the dependent input be updated based at least in part onthe value of the corresponding output at a time subsequent to thecorresponding output being generated during the running of the firsttest and prior to the dependent input being provided to the secondsubject application during the running of the second test.
 11. A methodas recited in claim 10, wherein performing the test comprises doing inthe order listed: performing the first test; updating the dependentinput based on the value of the corresponding output as generated duringthe running of the first test; and performing the second test
 12. Amethod as recited in claim 1, wherein performing the test comprisesautomatically setting a testing environment associated with the test toan initial condition associated with the test.
 13. A method as recitedin claim 1, wherein performing the test comprises automatically cleaningup a testing environment associated with the test to restore the testingenvironment at least in part to a condition in which it was prior to thetest being run.
 14. A method as recited in claim 1, wherein theapplication-specific data format comprises a complex data format.
 15. Amethod as recited in claim 1, wherein the application-specific dataformat comprises a custom complex data format.
 16. A method as recitedin claim 1, further comprising: receiving, in the event the subjectapplication fails one or more aspects of the test, one or more failurenotifications, each associated with a failure event; and associatingrelated ones of said one or more failure notifications, if any, togetherand assigning an error identification to each associated group offailure notifications and to each individual failure notification thatis not associated with any other failure notification.
 17. A method asrecited in claim 16, wherein the test involves a plurality of subjectapplications, each of which having an application-specific testconnector application associated with it, and further comprisingassociating each error identification that is specific to a particularsubject application with a subject application associated with thefailure notification(s) associated with the error identification.
 18. Amethod as recited in claim 17, wherein associating each erroridentification with a subject application comprises associating eacherror identification with an application-specific test connectorapplication associated with the failure notification(s) associated withthe error identification.
 19. A system for testing an application,comprising: a processor configured to: receive a definition of a test tobe performed on a subject application; and perform the test; and amemory configured to store the definition; wherein performing the testcomprises exchanging data with the subject application, as required toperform the test, using a connector application associated with thesubject application to do at least one of (1) convert an input data tobe supplied to the subject application during the test from a genericdata format not specific to the subject application into anapplication-specific data format associated with the subjectapplication, if the application-specific data format is different thanthe generic data format and (2) normalize an output data received fromthe subject application in the application-specific data format into thegeneric data format not specific to the subject application, if theapplication-specific data format is different than the generic dataformat.
 20. A computer program product for testing an application, thecomputer program product being embodied in a computer readable storagemedium and comprising computer instructions for: receiving a definitionof a test to be performed on a subject application; and performing thetest; wherein performing the test comprises exchanging data with thesubject application, as required to perform the test, using a testconnector application associated with the subject application to do atleast one of (1) convert an input data to be supplied to the subjectapplication during the test from a generic data format not specific tothe subject application into an application-specific data formatassociated with the subject application, if the application-specificdata format is different than the generic data format and (2) normalizean output data received from the subject application in theapplication-specific data format into the generic data format notspecific to the subject application, if the application-specific dataformat is different than the generic data format.